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ABSTRACT 


A critical limitation of most of the robots utilized in industrial environments arises due 
to their inability of utilize sensory feedback. This forces robot operation in a totally pre- 
programmed or teleoperation modes. In order to endow the new generation of robots with 
higher levels of autonomy techniques for sensing of their work environments and for accu- 
rate and efficient analysis of the sensory data must be developed. In this paper detailed 
development of vision system modules for inspecting various types of meters, both ana- 
log and digital, encountered in a robotic inspection and manipulation tasks are described. 
These modules are tested using industrial robot having multisensory input capability. 


1. Introduction 


Advanced robotic systems capable of performing a variety of tasks in complex un- 
structured environments will have to possess sophisticated sensory capability for acquiring 
information about their work environments. Also required is the associated capability 
for analyzing such information in an accurate and efficient manner. Robotic system with 
sophisticated sensory capability will be of particular utility in tasks such as automated as- 
sembly [5], inspection and manipulation in hazardous environments such as nuclear plants, 
[7] or space based platforms. Types of sensors which can be utilized in such systems in- 
clude vision, range, proximity, tactile, force and torque, etc. [3]. Of these, vision sensory 
modality is recognized as the one providing rich characterization of work environment with 
various types of relatively inexpensive and well engineered camera systems. 

The specific area of research reported in this paper deals with a vision system that 
has been designed and developed to perform various inspection and manipulation tasks 
associated with a control panel. The panel contains a number of displays, both analog 
and digital, and controls like switches and valves. In addition to the above panel the test 
bed for our research includes an industrial robot capable of sensing the environment with 
vision, range, proximity, force and torque and touch sensors. The main tasks performed 
by the vision system include: automatic location of the test panel, positioning of the robot 
arm to acquire appropriate input images of the panel, automatic recognition of all objects 
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appearing on the panel and determination of their 3-dimensional location. Once the ob- 
jects are located the system has to acquire finer resolution images of individual objects to 
determine their status. In this paper we shall present the detailed procedure developed to 
“read” various types of analog and digital meters. The paper includes results of several 
experiments carried out to verify the robustness of the system in performing automatic 
inspection and manipulation tasks. It is believed that the robot vision system can be 
utilized to perform tasks in a number of application domains including space. 


2. Vision Guided Robotic Inspection and Manipulation 


The main goal of a robot vision system is to provide an accurate interpretation of a 
scene utilizing images of the scene as the primary source of input. The interpretation can 
be provided in a variety of forms and at different levels of abstraction. A useful form 
of interpretation may include an object location map where different types of physical 
objects appearing in the scene are independently recognized and accurate locations of 
these objects in the scene are determined. Also, of utility is the information regarding the 
status or condition of an object. Design of a computer vision system that can perform 
such object recognition and scene interpretation is a complex and challenging task. The 
main difficulty underlying the task comes from the fact that images are 2-dimensional 
projections of the 3-dimensional real scene and innumerable combinations affecting the 
illumination source, scene and sensor parameters can result in the same observable value 
of recorded image intensity. 

In order to make the above problem computationally tractable model-based approach 
to computer vision is proposed [6]. The approach requires knowledge of a series of models 
associated with objects which are believed to appear in the scene. These models can be 
recorded in the knowledge-base of the system. Various features from the input images 
are extracted by using low- level, general purpose operators. These operators should be 
robust in extracting image features containing meaningful information about the objects. 
Finally, a correspondence is sought between the image derived features and scene domain 
models to recognize the objects. This is accomplished by utilizing various decision making 
schemes in the matching module. 

Development of autonomous systems for a variety of applications in an industrial en- 
vironment will require major research efforts to resolve many complex issues. We have 
undertaken an approach, which we believe allows making incremental progress towards 
the eventual development of such a system. Our initial research effort is directed towards 
researching issues associated with acquisition and analysis of multiple sensory data using 
a robotic system. This is accomplished by focusing on the development of an autonomous 
system that is capable of performing various inspection and manipulation tasks associated 
with a typical panel. For example these tasks can range from reading of various meters 
and displays, operating different types of switches and controls. Also, included are tasks 
associated with valve operation. Teleoperation or automatic operation of valves in nuclear 
power plants is recognized as one of the important desired capabilities of a robotic sys- 
tems. Design of the panel was done in consultation with a team from a nuclear power plant 
developer, using all “off-the-shelf” components. Our experimental set-up includes a test 
panel, a robot having multiple sensory capability, computers, and various manipulation 
tools. The test panel and the robot with various sensors mounted on the arm are shown 
in Figure 1. 

Typical autonomous robot operation will involve the following. The robot first identifies 
the exact geometrical position of the panel using a camera calibration program. Next it 
uses a computer vision system to develop an object location layout map for various devices 
appearing in the panel. The task command for the robot is provided through the binary 
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coded lights of an LCD display. After decoding the command the robot performs requested 
inspection or manipulation task. 

Robustness and ease in expandability to accomodate changes in the task environment 
are two key features guiding the development of the vision system. The system is compart- 
mentalized in two basic groups of procedures. The first group consists of general purpose 
procedures for camera calibration, image acquisition, knowledge acquisition, image seg- 
mentation, matching, and robot arm movements. The second group consists of special 
purpose procedures mainly designed for determining status of individual objects. 


3. Recognition of Object Status 


Depending upon the type and nature of the object the camera mounted on the arm is 
moved to take images using orthogonal viewing geometry. These images are analyzed to 
determine the status of the object. The objects appearing on the test panel and the type 
of status information associated with each one of them are listed in Table 1. 

Table 1 . List of objects appearing on the test panel and their status infor- 
mation. 


OBJECT TYPE 

STATUS 

1. Light 

On/Off 

2. Analog Meter 

Needle Reading 

3. Digital Meter 

7-Segment Code 

4. Valve 

Position of the Holes 

5. Slider Control 

Position of the Sliding Arm 

6. Push Button Switch 

On/Off 

7. Toggle Switch 

On/Off 


In order to account for changes in the task environment one will require an additional 
knowledge acquisition session to update the knowledge base and incorporation of the appro- 
priate routines to determine the status information of the objects added to the knowledge 
base. 

Status recognition of three object types are considered in this paper. They are two 
types of analog meters and digital (LCD) meter. Once the objects are identified by the 
object recognition module further processing is required to recognize their status. The 
primary requirements of the object status recognizers are their robustness, accuracy, and 
speed. Incorporation of limited apriori knowledge about the objects greatly facilitate in 
meeting these requirements. 

3.1 Reading an analog meter of type I 
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The main task in reading an analog meter is the determination of its needle orientation 
with respect to the horizontal direction. Once this angle is determined, the knowledge of 
the total swing angle of the needle for Full Scale Deflection(FSD) and the range of the 
meter enables one to compute the meter reading. It is assumed that the meter scale is 
symmetrical about the the vertical and that it is linear with respect to the angle. This 
assumption holds for many types of commercially available analog meters. 

In this section it is assumed that the analog meter has been isolated from the rest 
of the image by the segmentor, and all further processing is performed within a window 
containing only the analog meter. The approach consists of two major steps. 

The first step is the extraction of edges. For this, Marr edge detector [1] is chosen 
since it generates one pixel wide, closed contours. Since one can assume without loss of 
generality that the needle of a meter is a linear feature, the next step involves the analysis 
of all linear features within the processing window. For this, we choose Hough transforms 
[2]. The advantage of using Hough transforms includes it’s relative insensitivity to noise 
and to gaps in the image of a line. This makes the procedure less sensitive to the results 
of the edge detector. In the slope-intercept representation of a line, however, the Hough 
parameter space becomes unbounded due to the slope and intercept becoming infinite 
for vertical lines. Since a bounded parameter space is desired for the analysis, a normal 
representation of a line is often used. In this representation the parameters are p and 9 
and 

x cos 6 + y sin 6 = p (1) 

The parameter 9 can be considered to be bounded between -9\ and 9 2 . These angles are 
not required to be known exactly but the range needs to accomodate the total swing (i.e. 
all possible orientations) of the needle. Thus, it is reasonable to assume that this range of 
the parameter 9 is known for a particular analog meter. Also, parameter p is bounded by 
the length of the meter. This implies that the parameter p is bounded in the range 0 to 
length of the processing window, w, known from the object recognition module. Therefore 
the analysis of linear features only requires to be performed within the bounded parameter 
space - 9\ to 9 2 and 0 to w. 

In the step 2, following edge detection, the linear feature analysis is performed within 
the bounded space -9\ to 9 2 and 0 to w. An implication of this quantization range of 9 is 
that the horizontal linearities (9 = 90°) are not contained within the parameter space and 
therefore the horizontal edges are eliminated from further consideration. The elimination 
of false vertical linearities, linearities that does not correspond to the needle, is more 
involved since the needle itself may be vertically oriented. The key to the elimination of 
these false linearities is the observation that the vertical features that do not correspond 
to the needle have a p value which is either closer to 0 or w. Thus, seeking a local maxima, 
( 9 n ,p n ), in the parameter space away from p = 0, say p = 0 + A p , and away from p = w, 
say p = w — A p , where A p is a small number, will guarantee that the maxima is indeed 
due to the linearity of the needle. 

Figure 2 shows a sequence of processing steps for an image where the analog meter 
occupies less than 2% of the total area of the image. The robustness of the procedure is 
clearly demonstrated in this experiment. 

The accuracy of the final result is directly dependent on the accuracy of the angle of 
orientation of the needle. Hence, the accuracy of the result is dependent on the resolution 
of the parameter 9 in the parameter space. In the experiments shown in this section the 
parameter 9 was quantized to an 1 degree resolution. Since the analog meter had a reso- 
lution of 0.2 volts/degree this quantization results in a resolution of 0.2 Volts in the meter 
reading. This accuracy of the reading can be increased by finer division of the parameter 
9 at the cost of increasing execution time. . 

3.2 Reading an analog meter of type II 
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In this type of meters the needle is not pivoted at one end, but the needle moves 
across a horizontal scale. Thus, reading the meter requires determination of the needle 
position with respect to the left edge of the scale. Knowing the total length of the scale, L s , 
the range of the meter, x 0 to xj , and the needle position. l s , the meter reading. x r can be 
computed as: 


x r 


X f Xo 

L s 


h + Xq 


( 2 ) 


In practise however, the lengths L s and l, can not be derived from the images since 
the segmentor can not isolate the inner scale from the rest of the meter. This requires 
reformulation of equation (2) using the derivable distances w , length of the meter( window), 
and l w , the needle position with respect to the left edge of the meter(window). Using these 
distances x r can be found as, 


X f Xo * / r\\ 

x r = l w + x 0 - error (3) 

w 

This error term can be computed using an image reading a known value, say xo. In this 
case ,equation(2) will yield 

«/ “ *o, 

error = / 0 (4) 

w 

Since the needle is a linear vertical feature, the analysis of linear features applies to this 
task as well. The detection of the needle requires only a minor change in the linear feature 
analysis step of the section 3.1. Now the Hough parameter 6 is bounded between 0 — Ag 
and 0 + Ag, where A$ is a small angle. Notice that the 0 = 0° corresponds to a vertical 
line. The bounds of the parameter p remains the same as in section 3.1, i.e., between 0 
and w. This range of 9 eliminates the horizontal features and the false vertical features are 
eliminated using the same rule of section 3.1, i.e., the local maxima in parameter space is 
required to be away from p = 0 or p = w. 

Under these constraints, the local maxima ,(0„, p n ), will indeed correspond to the needle 
and l w = p n . Now we can compute the meter reading x r using equation(3), if the error 
term has been previously computed as explained by equation(4). 

The sequence of steps involved in processing an 128X128 image is illustrated in Figure 3. 
The original image is shown in Figure 3(a) and Figure 3(c) shows the line corresponding 
to the maxima in Hough space superimposed on the edge map (Figure 3(b)). These re- 
sults suggests that the presented procedure is an effective method for reading this type of 
analog meters, provided that the algorithm can be trained for computing the- error term 
in equation (3) using an image of known meter reading. The accuracy of the result is now 
dependent on the quantization of the parameter p. 

3.3 Reading a digital (LCD) meter 


This task actually consists of two recognition tasks. First, the digits needs to be 
identified, and secondly the decimal number represented by the individual digits needs to 
be identified. For recognition of digits we choose to use Fourier descriptors [4] for its size 
and rotational invariance properties. Once the individual digits are identified the decimal 
number is formed using an ad-hoc procedure explained in the next section. 

As in the previous sections we assume that the processing is performed within a window 
containing the digital meter. Since the digits are best discriminated using their edge 
properties, the first step in processing is the extraction of edges. In order to represent 
the structural shape of the objects in a suitable form for the Fourier descriptors, this 
step is followed by a thinning process. In this step we use a modification of an algorithm 
developed by Zhang and Suen [8], for skeletonizing the edges. This algorithm is known to 
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have some inherent drawbacks. One of the problems was due to the total elimination of 
small regions during the skeletonizing process. This limitation can however be overcome 
by simple modification. 

In order to use the Fourier descriptors to identify the digits we require the digit to 
appear as a single object, i.e. consisting of connected segments. However, it was observed 
that in the 7-segment display the vertical segments do not appear to be connected unless 
the middle horizontal segment is lit. Figure 4(a) shows the edge map of digits 0,1 and 7 in 
which the the digits do not appear as a single region due to the above. Therefore, the edge 
map was pre-processed before skeletonizing to fill the gaps between the vertical segments. 
Since we are primarily concerned with the breaks between vertical segments, the pixels 
labelled X and Y are required to be non-edge pixels. The mask is centered on non-edge 
pixels and if the number of edge pixels in the top part of the mask (P0 — P5) and the bottom 
part of the mask (P 6 — Pn) both exceeds a particular threshold (in our application we use 
a threshold of 1) then the center pixel was flagged to be an edge pixel. This procedure 
is performed asynchronously, i.e. the breaks are not filled until all the pixels have been 
considered. Figure 4(c) shows the results of performing the filling on Figure 4. The skeleton 
of Figure 4(c) is shown in (d). 

In step 4, we use Fourier descriptors to recognize the objects within the processing win- 
dow. This step uses prototypes of all 10 digits, pre-stored in a data base, for classification. 
Objects not matching any of the 10 prototypes to a higher degree is classified as unknown. 
In addition to recognizing the objects we also determine the minimum enclosing rectangle 
of each of the object. This information is used in identifying the position of the object 
within the window. Also computed is the area of the object. This is required to discrim- 
inate between the digit 0 and 8 since the Fourier descriptors are unable to discriminate 
between them. At the end of this step all objects within the processing window have been 
identified as a digit or an unknown. This completes the first task of identifying the digits. 

The next step is the formation of the decimal number from the individual digits iden- 
tified within the window. In step 5, the objects within the window is processed from 
the rightmost object to the leftmost object using the above rules where necessary. The 
rightmost object is determined using the coordinates of the minimum enclosing rectangle. 

Illustrated in Figure 4 are the sequence of processing steps for an 128X128 image. Since 
the recognition task is performed on edge maps of the image, for this procedure to perform 
error free, the edges needs to be generated correctly. This heavy dependence on the edge 
detector results limits the minimum size of the image. It was experimentally found that 
the LCD meter should occupy at least 15% of the total area to guarantee correct results. 


4. Concluding Remarks 


In this paper we describe development of modules associated with a robotic vision 
system for automatic inspection and manipulation tasks. The vision system consists of 
two groups of processing modules. The first comprises general purpose object recognition 
modules whereas the second comprises of specialized object status recogntion modules. 
Detailed development of modules for inspecting various types of meters, both analog and 
digital, is described. 
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Figure 1: The test-panel and an industrial robot with vision, range, touch, force, and 
proximity sensory capabilities. The test-panel includes variety of displays, meters, valves, 
controls, and switches. 
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Figure 2: Sequence of processing steps, (a) original image, (b) edge map (9x9 mask) of 
(a), (c) detected needle superimposed on (b). 
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Figure 3: Sequence of processing steps, (a) original image, (b) edge map (5x5 mask) of 
(a), (c) detected needle superimposed on (b). 
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Figure 4: Sequence of processing steps, (a) Typical edge maps of digits 0,1, and 7, (b) 
Results of filling, (c) Skeleton of (b) 
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